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PCON:  Motivation 


Semantic  integration  will  be 
one  of  the  first  major 
accomplishments  for 
ontology-based  applications 

-  Heterogeneous 
information  system  and 
resource  Interoperability 
is  a  major  concern  for 
military,  government, 
industry 

-  Many  view  this  as  the  a 
fundamental  technical 
challenge  of  the  Semantic 
Web 


To  answer  this  challenge, 
there  have  been  new 
developments  in  automated 
ontology  and  schema: 

o  Markup 

o  Alignment 

o  Merging 

o  Translation 

o  Learning 

Much  of  this  research  has 
been  funded  by  DARPA 
programs,  but  today  the 
largest  sponsors  are  EU 
programs 
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PCON:  Observation 


Semantic  integration 
research  community 
resembles  the  text  retrieval 
community  of  15  years  ago 

o  Critical  mass  of  globally 
distributed  research 
programs 

o  Large  variety  of  technical 
approaches 

o  Generally,  but  not 
universally,  accepted 
metrics 

o  No  meaningful  basis  of 
evaluating  one  technical 
approach  over  another 


•  The  success  of  text  retrieval 
technology  was  due  in  large 
measure  to  the  Text  Retrieval 
Conference  (TREC) 

-  Promoted  well-defined 
concepts  for  measuring 
success 

-  Clarified  metrics 

-  Established  realistic 
benchmarks 

-  Created  canonical 
challenge  problems 


The  NIST  TREC  model  has  a 
proven  record  of  successi 
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NIST  TREC  Model 


1.  Define  the  metrics 

2.  Develop  experiment  format  for  easy  participation 
by  researchers 

3.  Create  development  data  sets  and  test  data  sets; 
publish  the  former 

4.  Distribute  test  data  sets  to  experiment 
participants 

5.  Collect  automatically  generated  results  data 

6.  Collate  and  compare  results  data 

7.  Hold  assessment  workshop  and  end  of  cycle 
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PCON:  Timeline 


•  March  2004:  Met  with  NIST,  “pilot”  conference  as  PerMIS 
special  session  proposed 

•  March-June  2004: 

-  Formed  Organizational  Committee 

-  Recruited  participants 

-  Created  ontology  alignment  format 

-  Developed  test  ontology  pairs 

-  May  25:  Gave  presentation  at  DAML  PI  Meeting 

•  June  15  2004:  Released  test  ontology  pairs 

•  Juiy  16,  2004:  Coilected  alignment  results  data 

•  July  16-August  20,  2004:  Compiled  and  analyzed  results  data 

•  August  25,  2004:  PCON  special  session  at  PerMIS 


http://www.atl.lmco.com/projects/ontology/i3con.html 
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PCON  and  the  TREC  Model 


1.  Define  the  metrics 

2.  Develop  experiment  format 
for  easy  participation  by 
researchers 

3.  Create  development  data 
sets  and  test  data  sets; 
publish  the  former 

4.  Distribute  test  data  sets  to 
experiment  participants 

5.  Collect  automatically 
generated  results  data 

6.  Collate  and  compare 
results  data 

7.  Hold  assessment  workshop 
and  end  of  cycle 


^Precision,  Recall,  fMeasure 

^Ontology  Alignment 
Ontology;  Experiment  Set 
Platform 

^2  development  ontology 
pairs;  8  test  ontology  pairs 

-^5  participants 

^Most  participants  submitted 
alignment  data  for  all 
ontology  pairs 

^Where  we  are  today 
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I^CON:  ExperimentResults  Overview 


^ _ .■ _ I  ■! _ _ ' 


PCON:  Experiment  Results  Overview 


PCON  Experiment:  Lessons  Learned 


*  No  single  technical  approach  performed  best  on  all 
test  ontology  pairs 

*  No  single  ontology  pair  was  best  for  a// technical 
approaches 

*  All  approaches  performed  >0.5  fMeasure  on  at  least 
one  ontology  pair 

*  All  approaches  performed  <0.5  fMeasure  on  at  least 
one  ontology  pair 


There  is  much  more  to  be 
learned  from  the  I3CON 
experiment  data. 
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PCON:  Special  Thanks 

•  Organizational  Support 

-  Larry  Reeker  (NIST) 

-  Elena  Messina  (NIST) 

•  Technology  and  Data 

-  Ben  Ashpole  (ATL) 

-  Liz  Palmer  (ATL) 

-  Emil  Macarie  (ATL) 

-  Yun  Peng  (UMBC) 

-  Rong  Pan  (UMBC) 

•  Experiment  Participants 

-  Jerome  Pierson  (INRIA) 

-  John  Li  (Teknowledge) 

-  Lewis  Hart  (AT&T) 

-  Marc  Ehrig  (University  of 
Karlsruhe) 
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Guest  Speakers 

o  Bill  Andersen  (Ontology 
Works) 

o  Mike  Pool  (Information 
Extraction  and  Transport) 

o  Yun  Peng  (University  of 
Maryland  Baltimore 
County) 

o  Mike  Gruningner 

(University  of  Maryland) 
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EON  2004 


•  Evaluation  of  Ontology-based  Tools  3rd  International 
Workshop 

-  http://km.aifb.uni-karlsruhe.de/ws/eon2004/ 

•  Located  at  the  3rd  International  Semantic  Web  Conference 
(ISWC  2004) 

-  November  8,  2004 

-  Hiroshima  Prince  Hotel,  Hiroshima,  Japan 

•  EON  Ontology  Alignment  Experiment 

-  Provides  participants  with  a  complete  test  base  of 
ontology  pairs 

-  Test  is  based  on  one  particular  ontology  dedicated  to  a 
very  narrow  domain  and  a  number  of  alternative  ontologies 
of  the  same  domain 
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